Comparing DIF methods for data with dual dependency

Background During the past few decades, there have been many studies conducted to evaluate the comparative performance of differential item functioning (DIF) methods under various conditions. These conditions, for example, include small and unbalanced sample size between groups (Woods 2009), short tests (Paek and Wilson 2011), various levels of DIF contamination (Finch 2005), multilevel data (French and Finch 2010), violation of the normality assumption of latent traits (Woods 2011), and violation of the unidimensionality assumption (Lee et al. 2009). Among these conditions, violation of the local independence assumption has gained more attention recently, especially for large-scale assessments where local independence assumption is often violated. For example, the Trends in International Mathematics and Science Study (TIMSS) collected data from more than 60 countries worldwide in year 2011. Data collected from such an assessment, which consist of subdomains of a specific subject (e.g., algebra in the mathematics Abstract Background: The current study compared four differential item functioning (DIF) methods to examine their performancesin terms of accounting for dual dependency (i.e., person and item clustering effects) simultaneously by a simulation study, which is not sufficiently studied under the current DIF literature. The four methods compared are logistic regression accounting neither person nor item clustering effect, hierarchical logistic regression accounting for person clustering effect, the testlet model accounting for the item clustering effect, and the multilevel testlet model accounting for both person and item clustering effects. The secondary goal of the current study was to evaluate the trade-off between simple models and complex models for the accuracy of DIF detection. An empirical example analyzing the 2011 TIMSS Mathematics data was also included to demonstrate the differential performances of the four DIF methods. A number of DIF analyses have been done on the TIMSS data, and rarely had these analyses accounted for the dual dependence of the data. Results: Results indicated the complex models did not outperform simple models under certain conditions, especially when DIF parameters were considered in addition to significance tests. Conclusions: Results of the current study could provide supporting evidence for applied researchers in selecting the appropriate DIF methods under various conditions.